Display rack image processing device, image processing method, and recording medium

ABSTRACT

A state of a display rack is evaluated accurately. An image processing device includes a detection unit configured to detect a change area related to a display rack from a captured image in which an image of the display rack is captured, a classification unit configured to classify a change related to the display rack in the change area, and an evaluation unit configured to evaluate a display state of goods, based on a classification result.

This application is a National Stage Entry of PCT/JP2017/013678 filed onMar. 31, 2017, the contents of all of which are incorporated herein byreference, in their entirety.

TECHNICAL FIELD

The present disclosure relates to an image processing device, an imageprocessing method, and a recording medium.

BACKGROUND ART

When there is a deficiency in a display state of goods displayed on adisplay rack at a store such as a convenience store or a supermarket,that is, for example, when there is a shortage of goods displayed on thedisplay rack, a sales opportunity loss occurs, and sales at the store isheavily affected. Accordingly, when there is a deficiency in a displaystate of goods, it is preferable that goods replenishment work or thelike for eliminating the deficiency be promptly performed. Thus,monitoring of a display state of goods displayed on a display rack hasbeen sought.

For example, a device causing a clerk or the like to grasp work itemsfor the state monitoring area by evaluating a display state by use of aplurality of evaluation indicators related to disturbed display of goodsin a state monitoring area and presenting information about anevaluation result to the clerk is described (PTL 1).

Further, a background subtraction method of detecting a foreground area,based on background information of a photographed image, is described inPTL 2 and NPL 1.

CITATION LIST Patent Literature

-   [PTL 1] Japanese Unexamined Patent Application Publication No.    2016-207164-   [PTL 2] Japanese Unexamined Patent Application Publication No.    2008-176504

Non Patent Literature

-   [NPL 1] Zoran Zivkovic, “Improved Adaptive Gaussian Mixture Model    for Background Subtraction,” Proceedings of the 17th International    Conference on Pattern Recognition (ICPR 2004), U.S.A., IEEE Computer    Society, August, 2004, Volume 2, pp. 28 to 31

SUMMARY OF INVENTION Technical Problem

Types of change in a display rack include changes in a display rack suchas a change due to goods being taken by a customer, a change due togoods being replenished by a clerk, and a change in shape or appearancedue to a customer taking goods in his/her hand and returning the goodsto the original position. However, the technology described in PTL 1 islimited to evaluating a display rack by use of indicators related todisturbed display of goods and does not perform evaluation based on suchtypes of change in the display rack.

An object of the present disclosure is to provide a technology foraccurately evaluating a state of a display rack.

Solution to Problem

An image processing device according to an aspect of the presentdisclosure includes a detection means configured to detect a change arearelated to a display rack from a captured image in which an image of thedisplay rack is captured, a classification means configured to classifya change related to the display rack in the change area, and anevaluation means configured to evaluate a display state of goods, basedon a classification result.

Further, an image processing method according to an aspect of thepresent disclosure includes detecting a change area related to a displayrack from a captured image in which an image of the display rack iscaptured, classifying a change related to the display rack in the changearea, and evaluating a display state of goods, based on a classificationresult.

A computer program providing the aforementioned device or method by acomputer and a computer-readable non-transitory recording medium storingthe computer program also fall under the category of the presentdisclosure.

Advantageous Effects of Invention

The present disclosure is able to accurately evaluate a state of adisplay rack.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 shows a diagram illustrating a configuration example of goodsmonitoring system including an image processing device according to afirst example embodiment.

FIG. 2 shows a diagram for illustrating a use scene of the goodsmonitoring system.

FIG. 3 shows a functional block diagram illustrating an example of afunctional configuration of the image processing device according to thefirst example embodiment.

FIG. 4 shows a block diagram illustrating an example of a classificationdevice included in the image processing device according to the firstexample embodiment.

FIG. 5 shows a diagram for illustrating an operation of a detectionunit.

FIG. 6 shows a flowchart illustrating an example of an operation flow inthe image processing device according to the first example embodiment.

FIG. 7 shows a diagram illustrating an example of a classificationresult output by an area change classification unit.

FIG. 8 shows a diagram illustrating an example of display stateinformation stored in a second storage unit and an example of updateddisplay state information.

FIG. 9 shows a diagram for illustrating an operation of a calculationunit.

FIG. 10 shows a diagram illustrating an example of an output screendisplayed by an output device.

FIG. 11 shows a diagram illustrating another example of an output screendisplayed by the output device.

FIG. 12 shows a block diagram illustrating another example of theclassification device included in the image processing device accordingto the first example embodiment.

FIG. 13 shows a flowchart illustrating another example of an operationflow in the image processing device according to the first exampleembodiment.

FIG. 14 shows a block diagram illustrating another example of theclassification device included in the image processing device accordingto the first example embodiment.

FIG. 15 shows a flowchart illustrating another example of an operationflow in the image processing device according to the first exampleembodiment.

FIG. 16 shows a block diagram illustrating another example of theclassification device included in the image processing device accordingto the first example embodiment.

FIG. 17 shows a flowchart illustrating another example of an operationflow in the image processing device according to the first exampleembodiment.

FIG. 18 shows a block diagram illustrating another example of theclassification device included in the image processing device accordingto the first example embodiment.

FIG. 19 shows a flowchart illustrating another example of an operationflow in the image processing device according to the first exampleembodiment.

FIG. 20 shows a block diagram illustrating another example of theclassification device included in the image processing device accordingto the first example embodiment.

FIG. 21 shows a flowchart illustrating another example of an operationflow in the image processing device according to the first exampleembodiment.

FIG. 22 shows a diagram for illustrating an operation of a foregroundarea detection unit in a modified example.

FIG. 23 shows a diagram for illustrating an operation of the foregroundarea detection unit in the modified example.

FIG. 24 shows a diagram for illustrating an operation of the foregroundarea detection unit in the modified example.

FIG. 25 shows a functional block diagram illustrating an example of afunctional configuration of an image processing device according to asecond example embodiment.

FIG. 26 shows a flowchart illustrating an example of an operation flowin the image processing device according to the second exampleembodiment.

FIG. 27 shows a diagram exemplarily illustrating a hardwareconfiguration of a computer (information processing device) capable ofproviding each example embodiment of the present disclosure.

EXAMPLE EMBODIMENT First Example Embodiment

A first example embodiment of the present disclosure is described withreference to drawings. FIG. 1 shows a diagram illustrating aconfiguration example of goods monitoring system 1 including an imageprocessing device 100 according to the present example embodiment. FIG.2 shows a diagram for illustrating a use scene of the goods monitoringsystem 1.

As illustrated in FIG. 1, the goods monitoring system 1 includes theimage processing device 100, an image capturing device 2, and an outputdevice 3. The image processing device 100 is communicably connected tothe image capturing device 2 and the output device 3. While the imageprocessing device 100 is described on an assumption that the imageprocessing device 100 is configured to be separate from the imagecapturing device 2 and the output device 3, according to the presentexample embodiment, the image processing device 100 may be configured tobe built into the image capturing device 2 or may be configured to bebuilt into the output device 3. Further, there may be a plurality ofimage capturing devices 2. Further, an image captured by the imagecapturing device 2 may be a dynamic image or a series of static images.

For example, the output device 3 may be a display device such as adisplay or a point of sales system (POS) terminal. Further, withoutbeing limited to the above, the output device 3 may be a speaker or amobile terminal.

In the goods monitoring system 1, the image capturing device 2 capturesan image of a display rack 4 in a store, as illustrated in FIG. 2. Then,the image capturing device 2 transmits an image signal representing thecaptured image to the image processing device 100. For example, theimage capturing device 2 is a monitoring camera installed in the store.The image capturing device 2 may store the captured image inside theimage capturing device 2 or in a storage device different from the imageprocessing device 100.

A captured image acquired by the image capturing device 2 is describedon an assumption that the image is at least either of a color image(hereinafter referred to as a red green blue [RGB] image) and a distanceimage, according to the present example embodiment. For example, a colorimage may be an image in a color space other than an RGB image.

FIG. 3 shows a functional block diagram illustrating an example of afunctional configuration of the image processing device 100 according tothe present example embodiment. As illustrated in FIG. 3, the imageprocessing device 100 includes an acquisition unit 110, a detection unit120, a classification unit 130, a first storage unit 140, an evaluationunit 150, an output control unit 160, and a second storage unit 170. Theimage processing device 100 illustrated in FIG. 3 illustrates aconfiguration unique to the present disclosure, and it is needless tosay that the image processing device 100 illustrated in FIG. 3 mayinclude a component not illustrated in FIG. 3.

The acquisition unit 110 acquires an image signal representing acaptured image acquired by capturing an image of a display rack 4 by theimage capturing device 2. The acquisition unit 110 may receive an imagesignal transmitted from the image capturing device 2. The acquisitionunit 110 may acquire an image signal converted based on a captured imagestored inside the image capturing device 2 or in a storage devicedifferent from the image capturing device 2 and the image processingdevice 100. When the image processing device 100 is built into the imagecapturing device 2, the acquisition unit 110 may be configured toacquire a captured image itself.

The acquisition unit 110 converts an acquired image signal into an RGBimage and/or a distance image constituting the image signal and providesthe converted RGB image and/or distance image for the detection unit 120and the classification unit 130. The RGB image and/or the distance imageacquired by converting the image signal by the acquisition unit 110represents a captured image of the display rack 4 captured by the imagecapturing device 2 and therefore is also simply referred to as acaptured image.

The first storage unit 140 stores data used when processing by thedetection unit 120 and the classification unit 130 is performed. Datastored in the first storage unit 140 is described in a separate drawing.

The detection unit 120 detects a change area related to a display rack4. For example, when goods being included in a captured image and beingdisplayed on a display rack 4 is not included in an image (for example,a background image) acquired before the captured image, the detectionunit 120 detects an area of the goods. Further, for example, when goodsbeing included in a background image and being displayed on a displayrack 4 is not included in a captured image, the detection unit 120detects an area of the goods. Further, when goods being included in acaptured image and being displayed on a display rack 4, and the goodsincluded in a background image look differently, the detection unit 120detects an area of the goods. Further, when a captured image is capturedwhen a person or an object exists between a display rack 4 and the imagecapturing device 2, the detection unit 120 detects an area of the personor the object included in the captured image in which an image of thedisplay rack 4 is captured. Thus, the detection unit 120 detects achange area related to a display rack 4 such as a change area inside thedisplay rack 4 or a change area in a captured image caused by an objectbetween the display rack 4 and the image capturing device 2. Forexample, the detection unit 120 may generate a binary image having thesame size as a captured image and expressing a pixel value of a detectedchange area as 255 and the remaining area as 0. As a detection result ofa change area, the detection unit 120 provides a generated binary imagefor the classification unit 130. At this time, the detection unit 120may attach information indicating a captured image used in generation ofa binary image to the binary image and provide the binary image for theclassification unit 130, or may provide the captured image along withthe binary image for the classification unit 130.

A detection result has only to include information indicating a detectedchange area. For example, the detection unit 120 may associateinformation indicating a position of a detected change area (an areawith a pixel value 255) and a size of the change area with informationindicating a captured image and information indicating a backgroundimage that are used for detection of the change area, and output theassociated information as a detection result. Thus, a detection resultoutput by the detection unit 120 may take any form. An internalconfiguration of the detection unit 120 is described in a separatedrawing.

The classification unit 130 classifies a change related to a displayrack 4 in a change area. Based on a detection result (binary image)provided from the detection unit 120, and a previously learned changemodel related to the display rack 4 or distance information indicatingan image captured before an image capturing time of a captured image,the classification unit 130 classifies a change in a state of an imagein an area corresponding to the detected change area. For example, astate of an image includes a state in which goods is included or notincluded in an image, a state in which a customer is included or notincluded in an image, a state in which a shopping basket is included ornot included in an image, and a state in which a shopping cart isincluded or not included in an image. For example, the classificationunit 130 classifies a change related to a display rack 4 in a changearea as a change type such as “a change due to goods being no longerincluded on a display rack 4,” “a change due to goods being newlyincluded on a display rack 4,” “a change due to a change in appearanceof goods displayed on a display rack 4,” “a change due to existence of aperson in front of a display rack 4,” “a change due to existence of ashopping cart in front of a display rack 4,” or “a change due to achange in lighting.” The types for classifying a state change in achange area by the classification unit 130 are examples and types arenot limited to the above. Further, for example, “a change due to achange in appearance of goods displayed on a display rack 4” may beclassified in more detail into “a change in appearance due to a changeto a different goods” and “a change in appearance due to a change in aposition of goods.” An internal configuration of the classification unit130 is described in a separate drawing.

The classification unit 130 provides a classification result for theevaluation unit 150. While a classification result is specificallydescribed later, the result includes information indicating a changetype of a change area, information (referred to as change areainformation) indicating a rectangle circumscribed on the change area ona captured image acquired by the acquisition unit 110, a binary imagebeing a detection result of the change area output by the detection unit120, and information about a display rack 4 included in the capturedimage.

For example, change area information may be composed of x coordinatevalues and y coordinate values of four corners of a rectanglecircumscribed on an area of interest or may be composed of an xcoordinate and a y coordinate indicating at least one corner out of thefour corners of the circumscribed rectangle, and a width and a height ofthe circumscribed rectangle. Change area information is not limited toinformation indicating a rectangle, and may be information indicatinganother shape or may be information indicating an outline enclosing achange area.

Information about a display rack 4 included in a captured image may beinformation indicating a position where a captured image is captured,information indicating a position of the display rack 4, or anidentifier by which the display rack 4 can be specified.

The second storage unit 170 stores display state information 171 andmonitored area information 172. The second storage unit 170 may beprovided by a storage device different from the image processing device100 or may be built into the evaluation unit 150. Further, the secondstorage unit 170 and the first storage unit 140 may be provided in anintegrated manner. Further, the display state information 171 and themonitored area information 172 may be stored in separate storage units.

The display state information 171 indicates a display state of goods ona display rack 4. For example, the display state information 171 may bean image having the same size as a captured image acquired by theacquisition unit 110, the image being a binary image expressing a pixelvalue of an area where the goods exists as 255 and the remaining area as0. Further, for example, an initial value of the display stateinformation 171 may be previously given. The display state information171 includes information for specifying a display rack 4. Informationfor specifying a display rack 4 may be information indicating a positionof the image capturing device 2 capturing the display rack 4,information indicating a position of the display rack 4, or anidentifier by which the display rack 4 can be specified.

The monitored area information 172 indicates an area of a display rack 4being a monitoring target. For example, the monitored area information172 may be an image having the same size as a captured image acquired bythe acquisition unit 110 and being a binary image expressing a pixelvalue of an area of the display rack 4 being a monitoring target(referred to as monitoring target area) as 255 and the remaining area as0. Further, for example, there may be one or a plurality of monitoringtarget areas included in the monitored area information 172. Further,for example, the monitored area information 172 may be previously given.The monitored area information 172 includes information for specifying adisplay rack 4, similarly to the display state information 171.

The evaluation unit 150 calculates an amount of display being anevaluation indicator indicating a display state of goods on a displayrack 4, from a classification result and the display state information171 including information for specifying a display rack 4 related toinformation about the display rack 4 included in a captured imageincluded in the classification result. Specifically, the evaluation unit150 includes a display state update unit 151 and a calculation unit 153,as illustrated in FIG. 3.

The display state update unit 151 receives a classification result fromthe classification unit 130. As described above, a classification resultincludes information indicating a change type of a change area, changearea information, a detection result, and information about a displayrack 4 included in a captured image. The display state update unit 151specifies display state information 171 including information forspecifying a display rack 4 related to information about the displayrack 4 included in the captured image, in the display state information171 stored in the second storage unit 170. For example, when theinformation about the display rack 4 included in the captured image isinformation indicating a position where the captured image is captured,the display state update unit 151 specifies display state information171 including information indicating the same position. Further, forexample, when the information about the display rack 4 included in thecaptured image is information indicating a position of the display rack4, the display state update unit 151 specifies display state information171 including information indicating the same position. Further, forexample, when the information about the display rack 4 included in thecaptured image is an identifier by which the display rack 4 can bespecified, the display state update unit 151 specifies display stateinformation 171 including the same identifier.

The display state update unit 151 extracts, from the display stateinformation 171, an image of a part corresponding to a rectangular areaindicated by change area information included in a classificationresult. Then, the display state update unit 151 specifies, from theextracted image, a pixel corresponding to a change area indicated by adetection result included in the classification result. The displaystate update unit 151 updates a value of the specified pixel in thedisplay state information 171, based on information indicating a changetype of the change area included in the classification result. Forexample, the display state update unit 151 sets the value of the pixelto 0 when the change type is “a change due to goods being no longerincluded on a display rack 4,” and sets the value of the pixel to 255when the change type is “a change due to goods being newly included on adisplay rack 4,” and does not change the value of the pixel in the othercases.

The display state update unit 151 provides the updated display stateinformation 171 for the calculation unit 153. Further, the display stateupdate unit 151 stores the updated display state information 171 intothe second storage unit 170.

The calculation unit 153 receives updated display state information 171from the display state update unit 151. Further, the calculation unit153 acquires monitored area information 172 related to a display rack 4related to the updated display state information 171 from the secondstorage unit 170. Then, the calculation unit 153 calculates an amount ofdisplay of goods being an evaluation indicator indicating a displaystate of the display rack 4.

For each monitoring target area included in the monitored areainformation 172, the calculation unit 153 extracts an image of an areaof the updated display state information 171 corresponding to themonitoring target area. Then, the calculation unit 153 counts pixelshaving a pixel value 255 in the extracted image. The calculation unit153 calculates a size of each monitoring target area and calculates anamount of display for each monitoring target area with a number ofpixels having a pixel value 255 as a numerator and a size of themonitoring target area as a denominator. Then, the calculation unit 153provides the calculated amount of display for the output control unit160 along with information indicating the monitoring target area.Information indicating a monitoring target area is information includinginformation indicating a display rack 4 being a monitoring target andinformation indicating a position of the monitoring target area in themonitored area information 172.

The output control unit 160 receives an amount of display andinformation indicating a monitoring target area from the calculationunit 153. When the amount of display of the monitoring target area isless than or equal to a predetermined threshold value, that is, whenthere is a deficiency in a display state of goods, the output controlunit 160 transmits a control signal controlling the output device 3 tooutput information indicating existence of the deficiency. For example,when the output device 3 is a mobile terminal held by a clerk, theoutput control unit 160 transmits, to the mobile terminal, a controlsignal causing the mobile terminal to output existence of a deficiencyin a display state in a graspable manner for the clerk. The mobileterminal receiving the control signal may output information indicatingthe existence of the deficiency in a graspable manner for the clerk.Further, for example, when the output device 3 is a display device in abackyard of a store, the output control unit 160 transmits, to thedisplay device, a control signal causing the display device to outputexistence of a deficiency in a display state in a graspable manner for aclerk.

Thus, for example, a clerk at a store can readily grasp a display state.

Next, a configuration example of the acquisition unit 110, the detectionunit 120, the classification unit 130, and the first storage unit 140 isdescribed. A device including the acquisition unit 110, the detectionunit 120, the classification unit 130, and the first storage unit 140 ishereinafter referred to as a classification device 10.

Example 1 of Classification Device 10

FIG. 4 shows a block diagram illustrating an example of theclassification device 10 included in the image processing device 100according to the present example embodiment. As illustrated in FIG. 4,the classification device 10 includes an acquisition unit 110A, adetection unit 120A, a classification unit 130A, and a first storageunit 140A.

The acquisition unit 110A is an example of the acquisition unit 110. Theacquisition unit 110A acquires an image signal composed of an RGB image.The acquisition unit 110A converts the acquired image signal into an RGBimage constituting the image signal and provides the RGB image for thedetection unit 120A and the classification unit 130A.

The first storage unit 140A is an example of the first storage unit 140.The first storage unit 140A stores background information 141 and a rackchange model 142. The background information 141 is a reference imagefor making a comparison with a captured image in the detection unit 120Aand is also referred to as a background image. For example, it ispreferable that the background information 141 be the same type of imageas the captured image. As described above, a captured image is an RGBimage, according to the present example embodiment, and therefore it ispreferable that the background information 141 be also an RGB image. Thebackground information 141 may be a captured image provided first forthe detection unit 120A from the acquisition unit 110A or may be apreviously given image.

The rack change model 142 is a model modeling a previously learnedchange in a display rack 4. For example, the rack change model 142 maybe acquired by learning by use of machine learning such as a widelyknown convolutional neural network.

For example, a rack change model 142 represents “a change due to goodsbeing no longer included on a display rack 4” or “a change due to goodsbeing newly included on a display rack 4” learned by use of an image inwhich the display rack 4 includes goods and an image in which thedisplay rack 4 does not include goods. Further, a rack change model 142represents “a change due to a change in appearance of goods displayed ona display rack 4” learned by use of an image of a plurality of goodsgoods and a plurality of images in which a shape of each goods ischanged. Further, a rack change model 142 represents “a change due toexistence of a person in front of a display rack 4,” “a change due toexistence of a shopping cart in front of a display rack 4,” or the likelearned by use of a captured image captured in a state in which notarget exists in front of the display rack 4 and a captured imagecaptured in a state in which a target such as a person exists in frontof the display rack 4. Further, for example, a rack change model 142 mayrepresent “a change due to a change in lighting” learned by use ofimages in various environments.

Further for example, learning data of the rack change model 142 may be a6-channel image combining two RGB images before and after a change or a2-channel image combining any one of an R component, a G component, anda B component in each of two RGB image before and after a change.Further, for example, the learning data may be a 4-channel imagecombining any two of an R component, a G component, and a B component ineach of two RGB images before and after a change or a 2-channel imagecombining two RGB images before and after a change after conversion intogray-scale images. Further, the learning data may be an image combiningone or a plurality of channels in a color space after conversion intoanother color space such as a hue, saturation, and value (HSV) colorspace, RGB images before and after a change being converted into theother color space.

Further, the learning data of the rack change model 142 may be generatedfrom a color image such as an RGB image or may be generated by use ofboth a color image and a distance image.

As illustrated in FIG. 4, the detection unit 120A includes a foregroundarea detection unit 121 and a background information update unit 123.

The foreground area detection unit 121 receives a captured imageprovided from the acquisition unit 110A. Further, the foreground areadetection unit 121 acquires background information 141 related to thecaptured image from the first storage unit 140A. As described above, thebackground information 141 is an RGB image. The foreground areadetection unit 121 compares the two RGB images (the captured image andthe background information 141) and detects an area changing between thetwo RGB images as a change area. It can be said that the foreground areadetection unit 121 detects a foreground area in order to comparebackground information 141 being a background image with an RGB imagebeing a captured image.

A detection method of a change area by the foreground area detectionunit 121 is not particularly limited and may employ an existingtechnology. For example, the foreground area detection unit 121 maydetect a change area by use of the background subtraction methoddisclosed in NPL 1.

The foreground area detection unit 121 provides a generated binary imagefor the classification unit 130A as a detection result of a change area.At this time, the foreground area detection unit 121 may attach, to abinary image, information indicating a captured image used in generationof the binary image and information indicating the backgroundinformation 141, and provide the binary image for the classificationunit 130A or provide the captured image and the background information141 for the classification unit 130A along with the binary image.

Based on a captured image provided from the acquisition unit 110A and anRGB image being background information 141 stored in the first storageunit 140A, the background information update unit 123 updates thebackground information 141. An update method of background information141 by the background information update unit 123 is not particularlylimited and may employ, for example, a method similar to NPL 1.

An operation of the detection unit 120A is further described withreference to FIG. 5. FIG. 5 shows a diagram for illustrating anoperation of the detection unit 120A. A diagram (a) in FIG. 5 is anexample of a captured image, a diagram (b) in FIG. 5 is an example ofbackground information 141 related to the captured image, the backgroundinformation 141 being stored in the first storage unit 140A, and adiagram (c) in FIG. 5 shows a diagram illustrating an example of abinary image being a detection result of a change area.

The captured image and the background information 141 differ in areas ofgoods G1, goods G2, and goods G3. The goods G1 is not included in thebackground information 141 but is included in the captured image.Further, the goods G3 is included in the background information 141 butis not included in the captured image. Further, on the backgroundinformation 141, another goods is displayed at a position of the goodsG2 included in the captured image. Accordingly, the foreground areadetection unit 121 also detects the area of the goods G2 as an areaundergoing a change. Consequently, the foreground area detection unit121 generates a binary image in which the parts corresponding to theareas of the goods G1, the goods G2, and the goods G3 are represented inwhite, and the remaining part is represented in black, as illustrated inthe diagram (c) in FIG. 5.

In the following description, a change area refers to each white part inthe diagram (c) in FIG. 5. Specifically, for example, a change area is aset of pixels with a pixel value 255, a pixel value of at least one ofpixels adjacent to the pixel being 255. In the example in the diagram(c) in FIG. 5, the foreground area detection unit 121 detects threechange areas.

The classification unit 130A is an example of the classification unit130. As illustrated in FIG. 4, the classification unit 130A includes afirst extraction unit 131, a second extraction unit 132, and an areachange classification unit 134.

The first extraction unit 131 receives a binary image being a detectionresult from the foreground area detection unit 121. Further, the firstextraction unit 131 acquires a captured image used in generation of thebinary image from the first storage unit 140A. The first extraction unit131 may receive the captured image from the foreground area detectionunit 121 along with the binary image.

The first extraction unit 131 extracts an image of a change area from acaptured image. Specifically, by use of a captured image and a binaryimage having the same size as the captured image, the first extractionunit 131 extracts an image of an area on the captured imagecorresponding to an area with a pixel value 255 in the binary image as afirst image of interest. When the binary image is the diagram (c) inFIG. 5, the first extraction unit 131 extracts three first images ofinterest from the captured image. As described above, the captured imageis an RGB image, and therefore an extracted first image of interest isalso an RGB image.

For each change area, the first extraction unit 131 may extract a firstimage of interest in an area having the same shape as the change area ormay extract an image in an area enclosed by an outline having the sameshape as an outline in a predetermined shape and being circumscribed onthe change area, as a first image of interest. For example, a shape ofan outline circumscribed on the change area may be any shape such as arectangle or an ellipse. Further, the first extraction unit 131 mayextract an image in an area enclosed by an outline larger than anoutline circumscribed on the change area by a predetermined size, as afirst image of interest.

The first extraction unit 131 provides the extracted first image ofinterest for the area change classification unit 134. A area of a firstimage of interest extracted by the first extraction unit 131 on acaptured image is also referred to as a first area of interest.

The second extraction unit 132 receives a binary image being a detectionresult, from the foreground area detection unit 121. Further, the secondextraction unit 132 acquires background information 141 used ingeneration of the binary image from the first storage unit 140A. Thesecond extraction unit 132 may receive the background information 141from the foreground area detection unit 121 along with the binary image.

The second extraction unit 132 extracts an image of a change area frombackground information 141. Specifically, by use of backgroundinformation 141 being a background image and a binary image, the secondextraction unit 132 extracts an image of an area on the backgroundinformation 141 corresponding to an area with a pixel value 255 in thebinary image, as a second image of interest. An extraction method of asecond image of interest is similar to the extraction method of a firstimage of interest. The second extraction unit 132 provides the extractedsecond image of interest for the area change classification unit 134. Aarea of a second image of interest extracted by the second extractionunit 132 on background information 141 is also referred to as a secondarea of interest.

The area change classification unit 134 receives a first image ofinterest from the first extraction unit 131. Further, the area changeclassification unit 134 receives a second image of interest from thesecond extraction unit 132. Based on the rack change model 142 stored inthe first storage unit 140A, the area change classification unit 134classifies a change from a state of a second image of interest to astate of a first image of interest related to the second image ofinterest as, for example, a type described above. For example, based ona result of comparing a change from a state of a second image ofinterest to a state of a first image of interest with the rack changemodel 142, the area change classification unit 134 classifies thechange.

For example, the area change classification unit 134 may classify achange related to a display rack 4 as one of the aforementioned types byuse of a machine learning method (such as a convolutional neuralnetwork) by which the rack change model is created.

The area change classification unit 134 provides the classificationresult for the evaluation unit 150. Further, the area changeclassification unit 134 may store the classification result into, forexample, the first storage unit 140A.

Next, an operation flow of the image processing device 100 according tothe present example embodiment in this example is described withreference to FIG. 6. FIG. 6 shows a flowchart illustrating an example ofan operation flow in the image processing device 100 according to thepresent example embodiment.

First, the acquisition unit 110A acquires a captured image being an RGBimage, from an image signal in which an image of a display rack 4 iscaptured (Step S61). The acquisition unit 110A provides the acquiredcaptured image for the detection unit 120A and the classification unit130A.

Next, by use of the captured image being an RGB image provided from theacquisition unit 110A and background information 141 being an RGB imagestored in the first storage unit 140A, the foreground area detectionunit 121 in the detection unit 120A detects an area changing between thetwo RGB images as a change area (Step S62). Then, the foreground areadetection unit 121 provides the detection result of the change area forthe classification unit 130A. For example, the classification unit 130Agenerates a binary image in which a pixel in the detected change area isset to 255, and a pixel in the remaining area is set to 0, and providesthe binary image for the classification unit 130A as the detectionresult of the change area.

Further, the background information update unit 123 updates backgroundinformation 141 by use of the captured image and the backgroundinformation 141 (Step S63). Step S63 may be performed at any timingafter Step S61.

Based on the captured image provided from the acquisition unit 110A andthe detection result related to the captured image, the detection resultbeing provided from the foreground area detection unit 121, the firstextraction unit 131 in the classification unit 130A extracts an image ofan area (first area of interest) on the captured image corresponding toa change area indicated by the detection result, as a first image ofinterest (Step S64). The first extraction unit 131 provides theextracted first image of interest for the area change classificationunit 134.

Further, based on the detection result provided from the foreground areadetection unit 121 and the background information 141 used for acquiringthe detection result, the information being acquired from the firststorage unit 140A, the second extraction unit 132 in the classificationunit 130A extracts a second image of interest from the backgroundinformation 141 through an operation similar to that of the firstextraction unit 131 (Step S65). The second extraction unit 132 providesthe extracted second image of interest for the area changeclassification unit 134. Step S64 and Step S65 may be performedsimultaneously or may be performed in reverse order.

Then, based on the first image of interest provided from the firstextraction unit 131, the second image of interest provided from thesecond extraction unit 132, and the rack change model 142 stored in thefirst storage unit 140A, the area change classification unit 134classifies a change (a change from a state in the second image ofinterest to a state in the first image of interest) related to thedisplay rack 4 (Step S66).

An example of a classification result by the area change classificationunit 134 is illustrated in FIG. 7. FIG. 7 shows a diagram illustratingan example of a classification result output by the area changeclassification unit 134 in the classification unit 130A. For example,the area change classification unit 134 outputs a classification result70 as illustrated in FIG. 7.

As illustrated in FIG. 7, the classification result 70 includes a secondimage of interest 71, a first image of interest 72, a change type 73,change area information 74, a binary image identifier 75 indicating abinary image being a detection result of the change area, and displayrack information 76 related to a display rack 4 included in the capturedimage. For example, the binary image identifier 75 is an identifierindicating a binary image being output by the detection unit 120A andbeing stored in the first storage unit 140A or the like. For example,the display rack information 76 is an identifier by which a display rack4 can be specified. The classification result 70 illustrated in FIG. 7is an example, and the classification result 70 may include informationother than the information described in FIG. 7. For example, theclassification result 70 may include information (such as an identifierand an image capturing time) about the captured image or informationindicating a position of the first image of interest 72 in the capturedimage.

After Step S66 ends, the display state update unit 151 updates displaystate information 171 indicating a display state of goods, based on theclassification result (Step S67). Update processing of display stateinformation 171 by the display state update unit 151 is furtherdescribed with reference to FIG. 8. FIG. 8 shows a diagram illustratingan example of display state information 171 stored in the second storageunit 170 and an example of the updated display state information 171.

A detection result illustrated in a diagram (b) in FIG. 8 is the sameimage as the detection result illustrated in FIG. 5. In this example, animage related to the goods G3 illustrated in FIG. 5 is described. Thedisplay state update unit 151 extracts an image of a part correspondingto a rectangular area indicated by change area information included inthe classification result, from display state information 171 asillustrated in a diagram (a) in FIG. 8. It is assumed that the changearea information indicates a rectangle 81 in broken lines circumscribedon the goods G3 illustrated in the diagram (a) in FIG. 8.

From the extracted image, the display state update unit 151 specifies apixel corresponding to a change area indicated by the detection resultincluded in the classification result. Since a pixel value of the changearea is 255, the display state update unit 151 specifies a pixel in awhite part from an image of the detection result part of a rectangle 82corresponding to the rectangle 81.

Then, based on information indicating a change type of the change areaincluded in the classification result, the display state update unit 151updates the value of the specified pixel in the display stateinformation 171. For example, the display state update unit 151 sets thevalue of the pixel to 0 when the change type is “a change due to goodsbeing no longer included on a display rack 4,” sets the value of thepixel to 255 when the change type is “a change due to goods being newlyincluded on a display rack 4,” and does not change a pixel value in theother cases. Since the change in the goods G3 in the change area is “achange due to goods being no longer included on a display rack 4,” thedisplay state update unit 151 sets a value of a pixel in the specifiedchange area part out of pixels in the rectangle 81 in the display stateinformation 171 to 0. Then, the display state update unit 151 outputsupdated display state information 171 as illustrated in a diagram (c) inFIG. 8.

The display state update unit 151 stores the updated display stateinformation 171 into the second storage unit 170. Further, the displaystate update unit 151 provides the updated display state information 171for the calculation unit 153.

The calculation unit 153 calculates an amount of display of goods, basedon the updated display state information 171 provided from the displaystate update unit 151 and monitored area information 172 stored in thesecond storage unit 170 (Step S68). An operation of the calculation unit153 is further described with reference to FIG. 9. FIG. 9 shows adiagram for illustrating an operation of the calculation unit 153. Adiagram (a) in FIG. 9 illustrating an example of monitored areainformation 172 stored in the second storage unit 170. As illustrated inthe diagram (a) in FIG. 9, the monitored area information 172 includestwo monitoring target areas (91, 92). Further, a diagram (b) in FIG. 9is the updated display state information 171 illustrated in FIG. 8 onwhich outlines in broken lines representing the monitoring target areasare superposed. The calculation unit 153 counts pixels with a pixelvalue 255 in the updated display state information 171 included in themonitoring target area 91. Then, the calculation unit 153 calculates anamount of display with a size of the monitoring target area 91 includingthe goods G3 as a denominator and the counted number of pixels as anumerator.

Then, the output control unit 160 transmits a control signal based onthe calculation result to the output device 3 (Step S69). Then, theimage processing device 100 determines whether or not the acquisitionunit 110A receives a next image signal (whether or not a next capturedimage exists) (Step S70). When a next captured image exists (YES in StepS70), the processing proceeds to Step S61, and when a next capturedimage does not exist (NO in Step S70), the image processing device 100ends the operation.

FIG. 10 and FIG. 11 show diagrams illustrating examples of an outputscreen displayed by the output device 3. When the output device 3 is adisplay device, the output device 3 may output an output screen 101including a calculated amount of display as illustrated in FIG. 10,based on a control signal output from the output control unit 160.

Further, the output device 3 may output an output screen 111 asillustrated in FIG. 11, based on a control signal output from the outputcontrol unit 160. The output screen illustrated in FIG. 11 is a screenin which an amount of display calculated by the calculation unit 153 issuperimposed on the captured image illustrated in FIG. 5.

Further, the calculation unit 153 may store a calculated amount ofdisplay in, for example, the second storage unit 170 for a predeterminedperiod. At this time, the calculation unit 153 may store an amount ofdisplay for each monitoring target area in association with aclassification result. For example, it is assumed that a customertemporarily takes goods in his/her hand, moving to another place, andsubsequently returns the goods in his/her hand to the same rack withoutpurchasing the goods. Thus, when goods is not purchased, a POS cannotmanage an interest expressed in the goods by the customer. However, whena customer temporarily takes goods in his/her hand and moves to anotherplace, the detection unit 120A can detect that the goods disappears froma display rack 4, and therefore the calculation unit 153 calculates anamount of display lower than an amount of display before the customertakes out the goods from the display rack 4. Subsequently, when thegoods is returned to the display rack 4, the calculation unit 153calculates an amount of display with a higher value compared with anamount of display before the goods is returned to the display rack 4. Bystoring such a change in an amount of display for a predetermined periodand comparing the amount of display with sales data transmitted from aPOS, the calculation unit 153 may provide a place of goods in which thecustomer expresses interest for the output control unit 160.

Further, for example, it is assumed that a customer temporarily takesgoods in his/her hand and immediately returns the goods taken in his/herhand to the same rack. In this case, there is a high possibility that anappearance of the goods changes. However, an amount of display of thedisplay rack 4 does not change. However, the calculation unit 153 maystore an amount of display for each monitoring target area inassociation with a classification result and provide an amount ofdisplay in a predetermined period for the output control unit 160.

Consequently, for example, the output control unit 160 can output adegree of interest of a customer in goods or the like that cannot begrasped by a POS to the output device 3. Accordingly, the imageprocessing device 100 according to the present example embodiment canprovide data effective in marketing.

Further, for example, it is assumed that a customer places goods takenin his/her hand from a display rack 4 on a different display rack 4. Inthis case, the calculation unit 153 calculates an amount of display witha greater value than that of an amount of display before the goods isplaced. In such a case, the output control unit 160 may generate acontrol signal causing the output device 3 to output informationindicating a possibility that a different goods is placed, from anaccumulated amount of display and the calculated amount of display.

Further, the evaluation unit 150 may update the display stateinformation 171 to an initial state at a predetermined timing. Forexample, an initial state of the display state information 171 is astate generated from a previously created rack space allocation or thesame state as the background information 141. For example, apredetermined timing is a time when replenishment work of goods by aclerk is performed. By continuing updating the display state information171, an error from an actual display state may increase. However, byupdating the display state information 171 to an initial state at apredetermined timing by the evaluation unit 150, increase in an errorcan be prevented. Consequently, the evaluation unit 150 can preventoccurrence of an error in a calculated amount of display.

As described above, in the image processing device 100 according toexample 1 of the present example embodiment, the classification unit130A classifies a change related to a display rack 4 in a change arearelated to the display rack 4 detected from a captured image in which animage of the display rack 4 is captured, and the evaluation unit 150evaluates a display state of goods, based on the classification result.Since a change in the display rack 4 is classified as one of a pluralityof types by the classification unit 130A, the evaluation unit 150 canperform evaluation using thus classified result. Accordingly, theevaluation unit 150 can accurately evaluate a state of the display rack4.

Furthermore, in the classification device 10 in this example, thedetection unit 120A detects a change area related to a display rack 4 bycomparing a captured image in which an image of the display rack iscaptured with background information indicating an image captured beforean image capturing time of the captured image, and the classificationunit 130A classifies a change related to the display rack 4 in thechange area, based on a rack change model 142 being a previously learnedmodel of a change related to the display rack 4.

As described above, a rack change model 142 is a model representing achange related to a display rack 4, and therefore the classificationunit 130A classifies a change related to a display rack 4 in an areadetected as a change area as a type such as goods being taken out fromthe display rack 4 or goods being replenished.

Accordingly, the image processing device 100 including theclassification device 10 according to this example can specify not onlya change in goods on a display rack 4 but also the type of the change.Accordingly, the image processing device 100 can more accuratelydetermine a state of the display rack 4 such as a state in which goodsis taken out or a state in which the display rack 4 is replenished.

Since such a classification result tells whether goods displayed on adisplay rack 4 is goods to be purchased or goods taken in a hand, theimage processing device 100 including the classification device 10according to this example can output data effective in marketing.Further, since such a classification result tells that a customerpushing a cart or holding a shopping basket passes in front of a displayrack 4, or the like, the image processing device 100 can output, forexample, data usable for acquisition of a flow line of customers in astore.

Example 2 of Classification Device 10

FIG. 12 shows a block diagram illustrating another example of theclassification device 10 included in the image processing device 100according to the present example embodiment. As illustrated in FIG. 12,the classification device 10 includes an acquisition unit 110A, adetection unit 120B, a classification unit 130A, and a first storageunit 140B. The classification device 10 in this example tracks a changearea detected by a foreground area detection unit 221 between aplurality of RGB images. A component having the same function as acomponent included in the aforementioned drawing is given the samereference sign, and description thereof is omitted.

The detection unit 120B is an example of the detection unit 120.Further, the first storage unit 140B is an example of the first storageunit 140.

The first storage unit 140B stores background information 141 and a rackchange model 142, similarly to the first storage unit 140A. Further, thefirst storage unit 140B stores a detection result by the foreground areadetection unit 221 as foreground area information 243. The foregroundarea information 243 is described later.

The detection unit 120B includes a foreground area detection unit 221, abackground information update unit 223, and a foreground area trackingunit 224.

The foreground area detection unit 221 detects a change area through anoperation similar to the foreground area detection unit 121. Then, forexample, the foreground area detection unit 221 generates, as adetection result, a binary image expressing a pixel value of a detectedchange area as 255 and the remaining area as 0, similarly to theforeground area detection unit 121. Then, the foreground area detectionunit 221 associates the binary image being a detection result with animage capturing time of a captured image used in generation of thebinary image. The foreground area detection unit 221 provides thedetection result associated with the image capturing time of thecaptured image for the background information update unit 223 and theforeground area tracking unit 224. Further, the foreground areadetection unit 221 stores the detection result into the first storageunit 140B as foreground area information 243. In other words, foregroundarea information 243 is a binary image associated with an imagecapturing time of a captured image.

The foreground area tracking unit 224 tracks a change area detected bythe foreground area detection unit 221 between a plurality of capturedimages. The foreground area tracking unit 224 receives a detectionresult (binary image) provided from the foreground area detection unit221. Further, the foreground area tracking unit 224 acquires, from thefirst storage unit 140B, foreground area information 243 being a binaryimage generated from a captured image captured before an image capturingtime of a captured image related to a binary image being the detectionresult, the image capturing time being associated with the binary image.Then, by performing processing of correlating change areas representedby binary images with one another, the foreground area tracking unit 224tracks each change area. For example, the foreground area tracking unit224 may calculate a degree of similarity, based on at least one of asize, an shape, and an aspect ratio of a circumscribed rectangle of achange area represented by a binary image provided from the foregroundarea detection unit 221 and foreground area information 243 acquiredfrom the first storage unit 140B, and correlate change areas maximizingthe calculated degree of similarity with one another. Further, when theforeground area detection unit 221 is configured to extract colorinformation included in a detected change area from a captured image andassociate the acquired color information with the detection result, theforeground area tracking unit 224 may perform tracking by use of thecolor information. The foreground area detection unit 221 may associatea detection result with an image of a change area in place of colorinformation of the change area.

Then, when the tracking result is greater than or equal to apredetermined time, the foreground area tracking unit 224 provides thebinary image being the detection result provided from the foregroundarea detection unit 221 for the classification unit 130A. At this time,the foreground area tracking unit 224 may attach information indicatinga captured image used in generation of the binary image and informationindicating the background information 141 to the binary image andprovide the binary image for the classification unit 130A or may providethe captured image and the background information 141 for theclassification unit 130A along with the binary image. Further, when thebinary image includes a plurality of change areas and any of the changeareas is not tracked for a predetermined time or longer, the foregroundarea tracking unit 224 may provide the binary image for theclassification unit 130A along with information indicating the changearea being tracked for the predetermined time or longer.

Further, when the binary image includes a plurality of change areas, theforeground area tracking unit 224 may generate a plurality of binaryimages in such a way that one binary image includes one change area. Forexample, the foreground area tracking unit 224 may provide a binaryimage including only a change area tracked for a predetermined time orlonger for the classification unit 130A and discard a binary imageincluding a change area not tracked for the predetermined time orlonger. The foreground area tracking unit 224 may receive a binary imagefor each change area as a detection result from the foreground areadetection unit 221. A method of generating a binary image for eachchange area by the foreground area detection unit 221 is describedlater.

Further, the foreground area tracking unit 224 provides an update signalindicating update of background information 141 for the backgroundinformation update unit 223.

Further, for example, when an amount of movement of a change area isgreater than or equal to a predetermined threshold value, the foregroundarea tracking unit 224 may determine that an object included in thechange area is a moving body and discard the change area withoutproviding the change area for the classification unit 130A.Consequently, the image processing device 100 including theclassification device 10 in this example can delete a change related toa display rack 4 irrelevant to increase and decrease of goods, such as“a change due to existence of a person in front of a display rack 4” andtherefore can more accurately monitor a display state of goods. Theforeground area tracking unit 224 may provide a determination resultdetermining that an object included in a change area is a moving bodyfor the classification unit 130A, the determination result beingassociated with the change area. Then, when a determination result isassociated with a change area, the classification unit 130A may classifya change related to a display rack 4 in the change area as a typerelated to a change other than a change in goods displayed on thedisplay rack 4. For example, the classification unit 130A may classify achange related to a display rack 4 in the change area as a type relatedto a change other than a change in goods such as “a change due toexistence of a person in front of a display rack 4” or “a change due toexistence of a shopping cart in front of a display rack 4.”

Further, for example, when providing a detection result indicating achange area for the classification unit 130A after tracking the changearea, the foreground area tracking unit 224 may provide an update signalwith a value 1 for the background information update unit 223 along withinformation indicating the change area. Further, when not providing adetection result for the classification unit 130A, the foreground areatracking unit 224 may provide an update signal with a value 0 for thebackground information update unit 223 along with information indicatingthe change area. An update signal with a value 1 is an instructionindicating updating of an image of a part corresponding to a change areain background information 141, and an update signal with a value 0 is aninstruction indicating no updating of an image of a part correspondingto a change area in background information 141. For example, based on atracking time included in a tracking result, or, for example, purchaseinformation or stocking information of goods, work information of aclerk, or the like that are transmitted from a device external to theimage processing device 100, the foreground area tracking unit 224 mayoutput an update signal with a value 1 in such a way as to update abackground of a display rack 4 when there is a high possibility that thegoods included in a change area is purchased or replenished.

The background information update unit 223 updates backgroundinformation 141 through an operation similar to that of the backgroundinformation update unit 123, based on a captured image provided from theacquisition unit 110A, a detection result provided from the foregroundarea detection unit 221, the background information 141, and an updatesignal provided from the foreground area tracking unit 224.

The background information update unit 223 may not update, for example,an image of a part corresponding to a change area indicated by adetection result provided from the foreground area detection unit 221,in an RGB image indicated by background information 141. For example,when receiving the aforementioned update signal with a value 0 from theforeground area tracking unit 224, the background information updateunit 223 does not update background information of an area correspondingto the change area.

When not outputting a detection result to the classification unit 130A,the foreground area tracking unit 224 provides an update signal with avalue 0 for the background information update unit 223. For example, acase of not outputting a detection result to the classification unit 130refers to a case of a tracking result being less than a predeterminedtime or a case of an amount of movement of a change area being greaterthan or equal to a predetermined threshold value. Thus, when a trackingresult satisfies a first predetermined condition, the backgroundinformation update unit 223 receives an update signal with a value 0 anddoes not update background information of an area corresponding to thechange area. In other words, the background information update unit 223updates an area other than the area corresponding to the change area inthe background information 141. Consequently, an area in a capturedimage acquired by the acquisition unit 110A next, the area correspondingto an area not being updated, becomes more likely to be detected as achange area by the foreground area detection unit 221.

Further, for example, when a value of an update signal provided from theforeground area tracking unit 224 is 1, the background informationupdate unit 223 may update an image of a part corresponding to a changearea indicated by a detection result provided from the foreground areadetection unit 221, in an RGB image indicated by background information141. When a tracking result is greater than or equal to a predeterminedtime, the foreground area tracking unit 224 provides a detection resultindicating a tracked change area for the classification unit 130 andthen provides an update signal with a value 1 for the backgroundinformation update unit 223. In other words, when a tracking resultsatisfies a second predetermined condition that the result is a resulttracked for a predetermined time or longer, the background informationupdate unit 223 may receive an update signal with a value 1 from theforeground area tracking unit 224 and update an image of a partcorresponding to the change area in the background information 141.Consequently, the background information update unit 223 can bring thebackground information 141 stored in the first storage unit 140A closerto a captured image acquired by the acquisition unit 110A at that pointin time. Accordingly, the image processing device 100 including theclassification device 10 in this example can prevent an area on acaptured image acquired by the acquisition unit 110A next, the areacorresponding to the aforementioned change area, from being detected asa change area by the foreground area detection unit 221.

The classification unit 130A classifies a change related to a displayrack 4 through an operation described in example 1. At this time, whenreceiving both of a binary image being a detection result provided fromthe foreground area detection unit 221 and information indicating achange area tracked for a predetermined time or longer, the firstextraction unit 131 and the second extraction unit 132 may perform theextraction processing of a first image of interest and a second image ofinterest, respectively, on the change area tracked for the predeterminedtime or longer.

Next, an operation flow in the image processing device 100 including theclassification device 10 in this example is described with reference toFIG. 13. FIG. 13 shows a flowchart illustrating an example of anoperation flow in the image processing device 100 including theclassification device 10 in this example. Steps S131 and S132 describedin FIG. 13 are similar to S61 and S62 described in FIG. 6, respectively.

After Step S132 ends, the foreground area detection unit 221 storesforeground area information 243 into the first storage unit 140B (StepS133). As described above, the foreground area information 243 is adetection result associated with an image capturing time.

Next, the foreground area tracking unit 224 tracks the change area,based on the detection result provided from the foreground areadetection unit 221 and the foreground area information 243 (Step S134).The foreground area tracking unit 224 provides a binary image indicatinga change area tracked for a predetermined time or longer for theclassification unit 130A. The foreground area tracking unit 224 providesan update signal indicating updating of the background information 141for the background information update unit 223.

The background information update unit 223 updates the backgroundinformation 141, based on the captured image provided from theacquisition unit 110A, the detection result of the change area providedfrom the foreground area detection unit 221, the background information141, and the update signal provided from the foreground area trackingunit 224 (Step S135).

Step S135 may be performed simultaneously with or at an arbitrary timingafter Step S134.

Then, the image processing device 100 including the classificationdevice 10 in this example executes Steps S136 to S142 being processingsimilar to Steps S64 to S70 described in FIG. 6.

As described above, a change area detected by the detection unit 120B inthe image processing device 100 including the classification device 10in this example is tracked between a plurality of captured images, andthe classification unit 130A classifies a change related to a displayrack 4, based on the tracking result. For example, the detection unit120B provides the detection result for the classification unit 130A whenthe tracking result is greater than or equal to a predetermined time,and does not provide the detection result for the classification unit130A when the tracking result is less than the predetermined time.Accordingly, the classification unit 130A classifies a change related tothe display rack 4 with respect to a change area tracked for thepredetermined time or longer. Accordingly, an area where a change areais not continuously detected is not classified, and therefore anactually changing area can be accurately classified.

Further, for example, the detection unit 120B further provides thedetection result for the classification unit 130A when an amount ofmovement of the change area is less than a predetermined threshold valueand does not provide the detection result for the classification unit130A when the amount of movement is greater than or equal to thepredetermined threshold value. An object with an amount of movement inthe change area greater than or equal to the predetermined thresholdvalue is an object other than goods. Accordingly, a target of theclassification processing performed by the classification unit 130A canbe narrowed down to goods on a display rack 4, and thereforeclassification accuracy of goods on the display rack 4 can be furtherenhanced. Further, the image processing device 100 including theclassification device 10 in this example can prevent the classificationunit 130A from classifying a moving body such as a person as a change inthe display rack 4.

Example 3 of Classification Device 10

FIG. 14 shows a block diagram illustrating another example of theclassification device 10 included in the image processing device 100according to the present example embodiment. As illustrated in FIG. 14,the classification device 10 includes an acquisition unit 110C, adetection unit 120A, a classification unit 130C, and a first storageunit 140C. An image capturing device 2 included in goods monitoringsystem 1, according to the present example embodiment, may include aplurality of image capturing devices respectively acquiring differenttypes of images. For example, the image capturing device 2 may includean RGB camera acquiring an RGB image and a depth camera acquiring adistance image. In this case, the RGB camera and the depth camera areprovided at adjacent positions and capture images of the same target(display rack 4). Further, it is preferable that the RGB camera and thedepth camera be time synchronized and capture images of the display rack4 almost at the same time. Specifically, it is preferable that the depthcamera being a camera outputting a distance image in which an image ofan image capturing range of an RGB image captured by the RGB camera iscaptured within a predetermined time from an image capturing time of theRGB image. Further, the image capturing device 2 may be a sensor capableof acquiring a plurality of types of images (for example, an RGB imageand a distance image). For example, the image capturing device 2 may bean RGBD camera.

The classification device 10 in this example classifies a change relatedto a display rack 4 in a change area, based on a second captured imagebeing a distance image acquired by a second acquisition unit 312. Acomponent having the same function as a component included in theaforementioned drawing is given the same reference sign, and descriptionthereof is omitted.

The acquisition unit 110C includes a first acquisition unit 311 and thesecond acquisition unit 312.

The first acquisition unit 311 acquires a captured image being an RGBimage, similarly to the aforementioned acquisition unit 110A. A capturedimage being an RGB image acquired by the first acquisition unit 311 ishereinafter referred to as a first captured image.

The second acquisition unit 312 receives an image signal representing acaptured image acquired by capturing an image of a display rack 4 by theimage capturing device 2, similarly to the first acquisition unit 311,and acquires a distance image from the image signal. The secondacquisition unit 312 receives an image signal being a different typefrom an image signal acquired by the first acquisition unit 311. Forexample, when an image signal acquired by the first acquisition unit 311is an image signal constituting an RGB image, the second acquisitionunit 312 acquires an image signal constituting a distance image. Forexample, a distance image may refer to an image having a value of adistance from the image capturing device 2 to a target. Further, forexample, each pixel in a distance image may have a value in a range from0 to 255. At this time, for example, a value of each pixel, that is, adistance value may approach 0 as the target gets closer to the imagecapturing device 2 and may approach 255 as the target gets farther. Avalue of each pixel in a distance image is not limited to the above.This example is described on an assumption that a second captured imageacquired by the second acquisition unit 312 is a gray-scale distanceimage.

The second acquisition unit 312 may acquire an image signal convertedbased on a captured image stored inside the image capturing device 2 ora storage device different from the image capturing device 2 and theimage processing device 100, similarly to the first acquisition unit311. Further, when the image processing device 100 is built into theimage capturing device 2, the second acquisition unit 312 may beconfigured to acquire a captured image itself.

The second acquisition unit 312 converts an acquired image signal into adistance image constituting the image signal and provides the distanceimage for the classification unit 130C. A distance image acquired byconverting an image signal by the second acquisition unit 312 or acaptured image acquired from the image capturing device 2 is hereinafterreferred to as a second captured image.

The first acquisition unit 311 and the second acquisition unit 312 maybe integrally formed. Further, a first captured image and a secondcaptured image are associated with one another, based on informationindicating a position of image capture and an image capturing time.

The first storage unit 140C stores background information 141 similarlyto the first storage unit 140A and the first storage unit 140B. Further,the first storage unit 140C stores distance information 344. Thedistance information 344 is described later.

The classification unit 130C includes a first extraction unit 331, asecond extraction unit 332, an area change classification unit 334, anda distance information update unit 335.

The first extraction unit 331 extracts an image of a change area from asecond captured image. Specifically, by use of a second captured imagebeing a distance image provided from the second acquisition unit 312 anda binary image being a detection result provided from a foreground areadetection unit 121, the first extraction unit 331 extracts, as a firstimage of interest, an image of an area on the second captured imagecorresponding to an area with a pixel value 255 in the binary image. Thefirst extraction unit 331 may extract a first image of interest by amethod similar to that by the aforementioned first extraction unit 131.Then, the first extraction unit 331 provides the extracted first imageof interest for the area change classification unit 334.

The second extraction unit 332 extracts an image of a change area from adistance image captured before an image capturing time of a distanceimage being a second captured image associated with a first capturedimage used in generation of a binary image by the foreground areadetection unit 121. Specifically, the second extraction unit 332receives a binary image being a detection result from the foregroundarea detection unit 121. Further, the second extraction unit 332acquires, from the first storage unit 140C, distance information 344being a second captured image captured before an image capturing time ofa first captured image used in generation of the binary image. Distanceinformation 344 is a second captured image updated by the secondacquisition unit 312, to be described later, and is a distance imageacquired by the second acquisition unit 312. An image capturing time isassociated with distance information 344. As described above, a firstcaptured image and a second captured image are time synchronized, andtherefore an image capturing time of a first captured image and an imagecapturing time of a second captured image associated with the firstcaptured image are almost the same. Accordingly, it can be said that thesecond extraction unit 332 extracts an image of a change area, as asecond image of interest, from a (past) second captured image capturedbefore capture of a second captured image being a target of theextraction processing by the first extraction unit 331.

The second extraction unit 332 extracts a second image of interest by amethod similar to the method by which the first extraction unit 331extracts a first image of interest. The second extraction unit 332provides the extracted second image of interest for the area changeclassification unit 334.

The distance information update unit 335 updates distance information344, based on a distance image provided from the second acquisition unit312 and the distance information 344 stored in the first storage unit140C. For example, the distance information update unit 335 may updatedistance information 344 through an operation similar to that of thebackground information update unit 123.

The area change classification unit 334 classifies a change related to adisplay rack 4 in a change area, based on distance information in thechange area. First, the area change classification unit 334 receives afirst image of interest from the first extraction unit 331. Further, thearea change classification unit 334 receives a second image of interestfrom the second extraction unit 332. The area change classification unit334 classifies a change related to the display rack 4 in the changearea, based on the first image of interest and the second image ofinterest.

For example, the area change classification unit 334 may classify achange, based on an operation result acquired by subtracting a value(distance value) of each pixel in a second image of interest from avalue (distance value) of each pixel in a first image of interest. Forexample, when the operation result is a value greater than or equal to afirst predetermined threshold value, that is, when a target included inthe first image of interest is behind a target included in the secondimage of interest, the area change classification unit 334 may classifya change related to the display rack 4 in the change area as “a changedue to goods being no longer included on a display rack 4.” Further, forexample, when the operation result is a value less than or equal to asecond predetermined threshold value, that is, when a target included inthe first image of interest is closer to the image capturing device 2than a target included in the second image of interest, the area changeclassification unit 334 may classify a change related to the displayrack 4 in the change area as “a change due to goods being newly includedon a display rack 4.” Further, in the other cases, the area changeclassification unit 334 may classify a change related to the displayrack 4 in the change area as “a change due to a change in appearance ofgoods displayed on a display rack 4,” “a change due to a change inlighting,” or the like.

Further, for example, the area change classification unit 334 mayperform clustering on a distance value of each pixel in a first image ofinterest and a distance value of each pixel in a second image ofinterest, and set a distance value of a class with the maximum a numberof elements as a distance value representing each image of interest, andfurther determine coordinates of a cluster in the class of each image ofinterest. Then, for example, when an absolute value of a differencebetween the distance values representing the respective images ofinterest is less than or equal to a third predetermined threshold value,and also the coordinates of the clusters in the classes of therespective images of interest are apart by a fourth predeterminedthreshold value or greater, the area change classification unit 334 mayclassify a change related to the display rack 4 in the change area as “achange due to a change in appearance of goods displayed on a displayrack 4.”

Further, for example, the area change classification unit 334 mayclassify a change related to a display rack 4 in a change area as a typerelated to a change in goods displayed on the display rack 4 or a typerelated to a change due to an object other than goods displayed on thedisplay rack 4, by use of a result of subtracting a distance valuerepresenting a first image of interest from a previously set distancevalue from the image capturing device 2 to the display rack 4. Forexample, when the aforementioned result is a positive value, that is,when a target included in an image of a part in a change area in acaptured image exists between the image capturing device 2 and thedisplay rack 4, the area change classification unit 334 may classify achange related to the display rack 4 in the change area as a typerelated to a change due to an object other than goods displayed on thedisplay rack 4. For example, a type related to a change due to an objectother than goods displayed on the display rack 4 is at least one of “achange due to existence of a person in front of a display rack 4,” “achange due to existence of a shopping cart in front of a display rack4,” and “a change due to existence of a person and a shopping cart infront of a display rack 4.” Further, when the aforementioned result isnot a positive value, the area change classification unit 334 mayclassify a change related to the display rack 4 in the change area as atype related to a change in goods displayed on the display rack 4. Thus,by performing the classification processing by use of a first image ofinterest and a previously set distance value, the area changeclassification unit 334 can reduce a processing cost required for theclassification processing.

Next, an operation flow of the image processing device 100 including theclassification device 10 in this example is described with reference toFIG. 15. FIG. 15 shows a flowchart illustrating an example of anoperation flow in the image processing device 100 including theclassification device 10 in this example.

In Step S151 described in FIG. 15, a first captured image being an RGBimage is acquired from an image signal in which an image of a displayrack 4 is captured, similarly to S61 described FIG. 6 (Step S151).Further, the second acquisition unit 312 acquires a second capturedimage being a distance image from the image signal in which the image ofthe display rack 4 is captured (Step S152). An image capturing time ofthe second captured image has only to be within a predetermined timefrom an image capturing time of the first captured image, an imagecapturing range of the second captured image has only to be an imagecapturing range of the first captured image, and a timing of acquiringthe second captured image by the second acquisition unit 312 has only tobe before Step S155.

Then, the foreground area detection unit 121 detects a change area,similarly to Step S62 and Step S63 described in FIG. 6 (Step S153), andthe background information update unit 123 updates backgroundinformation 141 (Step S154).

Then, based on the second captured image provided from the secondacquisition unit 312 and a detection result provided from the foregroundarea detection unit 121, the first extraction unit 331 in theclassification unit 130C extracts, as a first image of interest, animage of an area (first area of interest) corresponding to a change areaindicated by the detection result on the second captured image (StepS155). The first extraction unit 331 provides the extracted first imageof interest for the area change classification unit 334.

Further, based on the detection result provided from the foreground areadetection unit 121 and distance information 344 indicating a secondcaptured image captured before an image capturing time of a secondcaptured image provided for the first extraction unit 331, the distanceinformation 344 being acquired from the first storage unit 140C, thesecond extraction unit 332 in the classification unit 130C extracts asecond image of interest from the distance information 344 through anoperation similar to that of the first extraction unit 331 (Step S156).The second extraction unit 332 provides the extracted second image ofinterest for the area change classification unit 334. Step S155 and StepS156 may be performed simultaneously or may be performed in reverseorder.

Then, based on a comparison result between a value of each pixel in thefirst image of interest and a value of each pixel in the second image ofinterest, the area change classification unit 334 classifies a changerelated to the display rack 4 in the change area (Step S157).

Next, based on the second captured image provided from the secondacquisition unit 312 and the distance information 344, the distanceinformation update unit 335 updates the distance information 344 (StepS158). Then, the classification device 10 performs processing similar toStep S67 to Step S69 described in FIG. 6 (Step S159 to Step S161).

Then, the image processing device 100 including the classificationdevice 10 in this example determines whether or not the firstacquisition unit 311 receives a next image signal and also the secondacquisition unit 312 receives a next image signal (whether or not a nextfirst captured image and a next second captured image exist) (StepS162). When a next first captured image and a next second captured imageexist (YES in Step S162), the processing proceeds to Step S151, and whenat least either of a next first captured image and a next secondcaptured image does not exist (NO in Step S162), the image processingdevice 100 including the classification device 10 in this example endsthe operation.

As described above, the detection unit 120A in the image processingdevice 100 including the classification device 10 in this exampledetects a change area by comparing a first captured image being an RGBimage with background information 141 indicating an image capturedbefore an image capturing time of the first captured image. Further, theclassification unit 130C in the image processing device 100 includingthe classification device 10 in this example classifies a change relatedto a display rack 4, based on a comparison result between a value ofeach pixel included in a second captured image being a distance imageand a value of each pixel in distance information 344 captured before animage capturing time of the second captured image.

Even with such a configuration, the image processing device 100 can moreaccurately determine a state of a display rack 4.

Example 4 of Classification Device 10

FIG. 16 shows a block diagram illustrating another example of theclassification device 10 included in the image processing device 100according to the present example embodiment. As illustrated in FIG. 16,the classification device 10 includes an acquisition unit 110C, adetection unit 120B, a classification unit 130D, and a first storageunit 140D.

In the classification device 10 in this example, the detection unit 120Bfurther includes a foreground area tracking unit 224 and tracks a changearea detected by a foreground area detection unit 221 between aplurality of RGB images. A component having the same function as acomponent included in the aforementioned drawing is given the samereference sign, and description thereof is omitted.

In this example, the foreground area tracking unit 224 may output anupdate signal to a distance information update unit 435.

The first storage unit 140D stores background information 141 anddistance information 344, similarly to the first storage unit 140C, andfurther stores foreground area information 243.

The classification unit 130D includes a first extraction unit 331, asecond extraction unit 332, an area change classification unit 334, anda distance information update unit 435. The distance information updateunit 435 updates distance information through an operation similar tothat of the distance information update unit 335.

Further, for example, the distance information update unit 435 may notupdate a part corresponding to a change area indicated by a binary imageprovided from the foreground area detection unit 221, in a distanceimage indicated by distance information 344. In other words, thedistance information update unit 435 may update a part other than a partcorresponding to a change area, in a distance image indicated bydistance information 344. By not updating distance information of a partcorresponding to a change area by the distance information update unit435, a difference in a part corresponding to the change area between asecond captured image acquired by the second acquisition unit 312 and adistance image captured before an image capturing time (at a past time)of the second captured image is clarified.

Further, for example, when a value of an update signal provided from theforeground area tracking unit 224 is 1, the distance information updateunit 435 may update a part corresponding to a change area indicated by adetection result provided from the foreground area detection unit 221,in a distance image indicated by distance information 344. In otherwords, the distance information update unit 435 may update distanceinformation of a part corresponding to a change area tracked by theforeground area tracking unit 224 in distance information 344 after adetection result (binary image) indicating the change area is providedfor the classification unit 130D. Consequently, the distance informationupdate unit 435 can bring the distance information 344 stored in thefirst storage unit 140D closer to a second captured image acquired bythe second acquisition unit 312 at that point in time. Accordingly, theimage processing device 100 including the classification device 10 inthis example can further enhance accuracy of a result of comparison ofdistance values by the area change classification unit 334 using an areaon a second captured image acquired by the second acquisition unit 312next, the area corresponding to a change area.

Next, an operation flow in the image processing device 100 including theclassification device 10 in this example is described with reference toFIG. 17. FIG. 17 shows a flowchart illustrating an example of anoperation flow in the image processing device 100 including theclassification device 10 in this example.

Step S171 to Step S173 described in FIG. 17 are processing similar toS151 to S153 described in FIG. 15, respectively.

After Step S173 ends, the foreground area detection unit 221 storesforeground area information 243 into the first storage unit 140D,similarly to aforementioned Step S133 (Step S174). Then, the foregroundarea tracking unit 224 tracks a change area, based on a detection resultprovided from the foreground area detection unit 221 and the foregroundarea information 243, similarly to aforementioned Step S134 (Step S175).The foreground area tracking unit 224 provides a binary image indicatinga change area tracked for a predetermined time or longer for theclassification unit 130D. The foreground area tracking unit 224 providesan update signal indicating updating of background information 141 andan update signal indicating updating of distance information 344 for thebackground information update unit 223 and the distance informationupdate unit 435, respectively.

The background information update unit 223 updates the backgroundinformation 141, based on the first captured image provided from thefirst acquisition unit 311, the detection result of the change areaprovided from the foreground area detection unit 221, the backgroundinformation 141, and the update signal provided from the foreground areatracking unit 224, similarly to aforementioned Step S135 (Step S176).

Subsequently, the classification device 10 performs processing similarto Step S155 to Step S157 described in FIG. 15 (Step S177 to Step S179).Then, the distance information update unit 435 updates the distanceinformation 344, based on the second captured image provided from thesecond acquisition unit 312, the distance information 344, and theupdate signal provided from the foreground area tracking unit 224 (StepS180).

Subsequently, the classification device 10 performs processing similarto Step S159 to Step S162 described in FIG. 15 (Step S181 to Step S184).

As described above, the classification device 10 in this example furtherincludes the foreground area tracking unit 224 described in example 2 inthe classification device 10 in example 3. Even with such aconfiguration, the image processing device 100 including theclassification device 10 in this example can accurately classify anactually changing area.

Example 5 of Classification Device 10

FIG. 18 shows a block diagram illustrating another example of theclassification device 10 included in the image processing device 100according to the present example embodiment. Aforementioned example 1 toexample 4 are described with an example of a captured image input to thedetection unit 120A or the detection unit 120B being an RGB image;however, an image input to the detection unit may be a distance image.The classification device 10 in this example is described on anassumption that a distance image is input to a detection unit. Acomponent having the same function as a component included in theaforementioned drawing is given the same reference sign, and descriptionthereof is omitted.

As illustrated in FIG. 18, the classification device 10 in this exampleincludes an acquisition unit 110E, a detection unit 120E, aclassification unit 130E, and a first storage unit 140E.

The acquisition unit 110E acquires a captured image being a distanceimage, similarly to the aforementioned second acquisition unit 312. Theacquisition unit 110E provides the acquired captured image for thedetection unit 120E and the classification unit 130E.

The first storage unit 140E stores background information 541. Thebackground information 541 is a reference image for making a comparisonwith a captured image in the detection unit 120E and is also referred toas a background image. As described above, a captured image is adistance image in this example. Accordingly, it is preferable that thebackground information 541 be a distance image being the same type ofimage as a captured image. The background information 541 may be acaptured image provided first for the detection unit 120E from theacquisition unit 110E or may be a previously given image. The backgroundinformation 541 is similar to the aforementioned distance information344.

The detection unit 120E includes a foreground area detection unit 521and a background information update unit 523. The foreground areadetection unit 521 receives a captured image provided from theacquisition unit 110E. Further, the foreground area detection unit 521acquires background information 541 related to a captured image from thefirst storage unit 140E. The foreground area detection unit 521 detectsan area changing between two distance images as a change area(foreground area). For example, the foreground area detection unit 521may detect a change area through an operation similar to the foregroundarea detection unit 121. The foreground area detection unit 521generates, as a detection result of a change area, a binary imageexpressing a pixel value of the detected change area as 255 and theremaining area as 0, similarly to the foreground area detection unit121, and provides the generated binary image for the classification unit130E.

The background information update unit 523 updates backgroundinformation 541, based on a captured image provided from the acquisitionunit 110E and a distance image being background information 541 storedin the first storage unit 140E. For example, the background informationupdate unit 523 may update background information 541 through anoperation similar to that of the background information update unit 123.

The classification unit 130E includes a first extraction unit 531, asecond extraction unit 532, and an area change classification unit 334.

The first extraction unit 531 extracts a first image of interest from acaptured image, similarly to the first extraction unit 331. Then, thefirst extraction unit 531 provides the extracted first area of interestfor the area change classification unit 334.

The second extraction unit 532 extracts an image of a change area fromthe background information 541 as a second image of interest. Anextraction method of a second image of interest by the second extractionunit 532 is similar to that by the second extraction unit 332.

The area change classification unit 334 classifies a change related to adisplay rack 4 in a change area, based on distance information in thechange area, similarly to the area change classification unit 334described in example 3.

Next, an operation flow of the image processing device 100 including theclassification device 10 in this example is described with reference toFIG. 19. FIG. 19 shows a flowchart illustrating an example of anoperation flow in the image processing device 100 including theclassification device 10 in this example.

First, the acquisition unit 110E acquires a captured image being adistance image from an image signal in which a display rack 4 iscaptured (Step S191). The acquisition unit 110E provides the acquiredcaptured image for the detection unit 120E and the classification unit130E.

Next, by use of the captured image being a distance image provided fromthe acquisition unit 110E and background information 541 being adistance image stored in the first storage unit 140E, the foregroundarea detection unit 521 in the detection unit 120E detects an areachanging between the two distance images as a change area (Step S192).Then, the foreground area detection unit 521 provides the detectionresult of the change area for the classification unit 130E.

Further, the background information update unit 523 updates thebackground information 541 by use of the captured image and thebackground information 541 (Step S193). Step S193 may be performed atany timing after Step S191.

Based on the captured image provided from the acquisition unit 110E andthe detection result being related to the captured image and beingprovided from the foreground area detection unit 521, the firstextraction unit 531 in the classification unit 130E extracts an image ofan area (first area of interest) corresponding to a change areaindicated by the detection result on the captured image, as a firstimage of interest (Step S194). The first extraction unit 531 providesthe extracted first image of interest for the area change classificationunit 334.

Further, based on the detection result provided from the foreground areadetection unit 521 and the background information 541 being used foracquiring the detection result and being acquired from the first storageunit 140E, the second extraction unit 532 in the classification unit130E extracts a second image of interest from the background information541 through an operation similar to that of the first extraction unit531 (Step S195). The second extraction unit 532 provides the extractedsecond image of interest for the area change classification unit 334.Step S194 and Step S195 may be performed simultaneously or may beperformed in reverse order.

Then, based on the first image of interest provided from the firstextraction unit 531 and the second image of interest provided from thesecond extraction unit 532, the area change classification unit 334classifies a change (a change from a state in the second image ofinterest to a state in the first image of interest) related to thedisplay rack 4 (Step S196).

Then, the classification device 10 performs processing similar to StepS67 to Step S69 described in FIG. 6 (Step S197 to Step S199). Then, theimage processing device 100 including the classification device 10determines whether or not the acquisition unit 110E receives a nextimage signal (whether or not a next captured image exists) (Step S200).When a next captured image exists (YES in Step S200), the processingproceeds to Step S191, and when a next captured image does not exist (NOin Step S200), the image processing device 100 ends the operation.

As described above, the detection unit 120E in the classification device10 in this example detects a change area by comparing a captured imagebeing a distance image with background information 541 indicating animage captured before an image capturing time of the captured image.Then, based on the comparison result between a value of each pixelincluded in the captured image and a value of each pixel in thebackground information 541, the classification unit 130E classifies achange related to the display rack 4.

Even with such a configuration, the image processing device 100including the classification device 10 in this example can moreaccurately determine a state of a display rack 4.

Example 6 of Classification Device 10

FIG. 20 shows a block diagram illustrating another example of theclassification device 10 included in the image processing device 100according to the present example embodiment. The image processing device100 including the classification device 10 in this example tracks achange area detected by a foreground area detection unit 621 between aplurality of distance images. A component having the same function as acomponent included in the aforementioned drawing is given the samereference sign, and description thereof is omitted.

As illustrated in FIG. 20, the classification device 10 includes anacquisition unit 110E, a detection unit 120F, a classification unit130E, and a first storage unit 140F.

The first storage unit 140F stores background information 541, similarlyto the first storage unit 140E. In addition, the first storage unit 140Fstores foreground area information 243.

The detection unit 120F includes the foreground area detection unit 621,a background information update unit 623, and a foreground area trackingunit 224.

The foreground area detection unit 621 detects a change area through anoperation similar to the foreground area detection unit 521. Then, forexample, the foreground area detection unit 621 generates, as adetection result, a binary image expressing a pixel value of thedetected change area as 255 and the remaining area as 0, similarly tothe foreground area detection unit 521. Then, the foreground areadetection unit 621 associates a binary image being the detection resultwith an image capturing time of a captured image used in generation ofthe binary image. The foreground area detection unit 621 provides thedetection result associated with the image capturing time of thecaptured image for the background information update unit 623 and theforeground area tracking unit 224. Further, the foreground areadetection unit 621 stores the detection result into the first storageunit 140F as foreground area information 243.

The background information update unit 623 updates backgroundinformation 541 through an operation similar to the backgroundinformation update unit 523, based on a captured image provided from theacquisition unit 110E, a detection result provided from the foregroundarea detection unit 621, the background information 541, and an updatesignal provided from the foreground area tracking unit 224. Thebackground information update unit 623 may or may not update an image ofa part corresponding to a change area, similarly to the backgroundinformation update unit 223.

Further, the foreground area tracking unit 224 tracks a change area byuse of binary images generated from distance images, similarly to theforeground area tracking unit 224 in example 2.

Next, an operation flow in the image processing device 100 including theclassification device 10 in this example is described with reference toFIG. 21. FIG. 21 shows a flowchart illustrating an example of anoperation flow in the image processing device 100 including theclassification device 10 in this example. Steps S211 and S212 describedin FIG. 21 are similar to S191 and S192 described in FIG. 19,respectively.

After Step S212 ends, the foreground area detection unit 621 storesforeground area information 243 into the first storage unit 140F (StepS213). As described above, the foreground area information 243 is adetection result associated with an image capturing time.

Next, based on the detection result provided from the foreground areadetection unit 621 and the foreground area information 243, theforeground area tracking unit 224 tracks a change area (Step S214). Theforeground area tracking unit 224 provides a binary image indicating achange area tracked for a predetermined time or longer for theclassification unit 130E. The foreground area tracking unit 224 providesan update signal indicating updating of the background information 541for the background information update unit 623.

Based on the captured image provided from the acquisition unit 110E, thedetection result of the change area provided from the foreground areadetection unit 621, the background information 541, and the updatesignal provided from the foreground area tracking unit 224, thebackground information update unit 623 updates the backgroundinformation 541 (Step S215).

Step S215 may be performed simultaneously with or at an arbitrary timingafter Step S214.

Then, the image processing device 100 including the classificationdevice 10 in this example executes Steps S216 to S222 being processingsimilar to Steps S194 to S200 described in FIG. 19.

As described above, the classification device 10 in this example furtherincludes the foreground area tracking unit 224 in example 2 in theclassification device 10 in example 5. Even with such a configuration,the image processing device 100 including the classification device 10in this example can accurately classify an actually changing area.

Modified Example of Foreground Area Detection Unit

The foreground area detection unit (121, 221, 521, 621) included in theclassification device 10 in each of the aforementioned examples mayspecify that a target included in a change area is a target other thangoods in a display rack 4, by further using preregistered rack areainformation.

A modified example of the foreground area detection unit 121 in theclassification device 10 in example 1 is described in this modifiedexample; however, this modified example is also applicable to theforeground area detection unit in each of example 2 to example 6.

FIG. 22 to FIG. 24 show diagrams for illustrating an operation of aforeground area detection unit 121 in this modified example.

It is assumed that the foreground area detection unit 121 detects achange area by comparing a captured image provided from an acquisitionunit 110A with background information 141 and generates, for example, adetection result 21 being a binary image indicating the change area asillustrated in FIG. 22. It is further assumed that the detection resultincludes three change areas being a change area 22, a change area 23,and a change area 24. The foreground area detection unit 121 generates adetection result 21A, a detection result 21B, and a detection result 21Cbeing separate binary images for the respective change areas, byapplying a common labeling method to the detection result 21.

In other words, when a detection result includes a plurality of changeareas, the foreground area detection unit 121 generates a plurality ofbinary images in such a way that the respective change areas areincluded in separate binary images.

Then, based on preregistered rack area information and each of theplurality of binary images, the foreground area detection unit 121determines whether or not a change area is an area where a changerelated to a change in goods is detected.

The rack area information indicates an area where goods is displayed ina display rack 4. Since the goods monitoring system 1 monitors goods ona display rack 4, an area where the goods is displayed, the area beingindicated by the rack area information, is also referred to as amonitoring target area, and the rack area information is also referredto as monitored area information. For example, the rack area informationmay be an image having the same size as a captured image acquired by theacquisition unit 110A and being a binary image expressing a pixel valueof a monitoring target area of a display rack 4 being a monitoringtarget as 255 and the remaining area as 0. Further, for example, theremay be one or a plurality of monitoring target areas included in therack area information. For example, the rack area information may bepreviously stored in a first storage unit 140A. The rack areainformation includes information for specifying a display rack 4included in a captured image acquired by the acquisition unit 110A.

For example, by use of rack area information 25 related to a displayrack 4 included in the captured image acquired by the acquisition unit110A, as illustrated in FIG. 23, the foreground area detection unit 121performs a logical conjunction operation with the detection result 21A,the detection result 21B, or the detection result 21C for eachcorresponding pixel. Since a monitoring target area is represented inwhite in the rack area information 25, as illustrated in FIG. 23, therack area information 25 includes six monitoring target areas.

An operation result 26A illustrated in FIG. 24 is a result of thelogical conjunction operation on the rack area information 25 and thedetection result 21A, an operation result 26B is a result of the logicalconjunction operation on the rack area information 25 and the detectionresult 21B, and an operation result 26C is a result of the logicalconjunction operation on the rack area information 25 and the detectionresult 21C.

An object other than goods, such as a person or a cart, extends over aplurality of rack areas, and therefore as a result of the logicalconjunction operation on the detection result 21A and the rack areainformation 25, a part (white part) with a pixel value 255 indicating achange area is divided into a plurality of areas as is the case with theoperation result 26A illustrated on the left side of FIG. 24. On theother hand, a part (white part) indicating a change area in each of theoperation result 26B and the operation result 26C does not change fromeach of the detection result 21B and the detection result 21C,respectively, and is a continuous area (a set of pixels with a pixelvalue 255, at least one of pixels adjacent to each pixel being a pixelwith a pixel value 255). A goods displayed in a display area (monitoringtarget area) of a display rack 4 does not extend over a plurality ofmonitoring target areas. Accordingly, when a change area is divided intoa plurality of areas as is the case with the operation result 26A, theforeground area detection unit 121 determines that a change to thechange area is a change not caused by goods and does not include thechange in a detection result provided for the classification unit 130A.In other words, the foreground area detection unit 121 provides thedetection result 21B and the detection result 21C for the classificationunit 130A.

Consequently, the classification unit 130A can perform theclassification processing on a change to goods displayed on a displayrack 4 and therefore can prevent degradation in classification accuracyof a change to goods due to an effect of an object other than goods.Further, when a change in a change area is a change due to an objectother than goods, the foreground area detection unit 121 can make aclassification before the classification unit 130A performs theclassification processing, and therefore an amount of processing by theclassification unit 130A can be reduced.

Second Example Embodiment

A second example embodiment of the present disclosure is described withreference to drawings. A minimum configuration according to the presentexample embodiment for resolving the problem to be resolved by thepresent disclosure is described.

FIG. 25 shows a functional block diagram illustrating an example of afunctional configuration of an image processing device 250 according tothe present example embodiment. As illustrated in FIG. 25, the imageprocessing device 250 includes a detection unit 251, a classificationunit 252, and an evaluation unit 253.

The detection unit 251 has the function of the detection unit 120according to the aforementioned first example embodiment. The detectionunit 251 detects a change area related to a display rack from a capturedimage in which an image of the display rack is captured. For example,the detection unit 251 detects a change area by comparing a capturedimage with background information indicating an image captured before animage capturing time of the captured image. The detection unit 251provides information indicating the detected change area for theclassification unit 252.

The classification unit 252 has the function of the classification unit130 according to the aforementioned first example embodiment. Theclassification unit 252 classifies a change related to a display rack ina change area. For example, the classification unit 252 classifies achange related to a display rack in a change area, based on a previouslylearned model of a change related to the display rack or distanceinformation indicating an image captured before an image capturing timeof a captured image. For example, by comparing a previously learnedmodel of a change related to a display rack with a change in a detectedchange area, the classification unit 252 classifies a change in thechange area as one of a plurality of change types. For example, changetypes include “a change due to goods being no longer included on adisplay rack,” “a change due to goods being newly included on a displayrack,” “a change due to a change in appearance of goods displayed on adisplay rack,” “a change due to existence of a person in front of adisplay rack,” “a change due to existence of a shopping cart in front ofa display rack,” and “a change due to a change in lighting.” Theclassification unit 252 provides the classification result for theevaluation unit 253.

The evaluation unit 253 has the function of the evaluation unit 150according to the aforementioned first example embodiment. The evaluationunit 253 evaluates a display state of goods, based on a classificationresult.

FIG. 26 shows a flowchart illustrating an operation example of the imageprocessing device 250 according to the present example embodiment. Thedetection unit 251 detects a change area related to a display rack froma captured image in which the display rack is captured (Step S261).

Then, the classification unit 252 classifies a change related to thedisplay rack in the change area (Step S262).

Subsequently, the evaluation unit 253 evaluates a display state ofgoods, based on the classification result (Step S263).

As described above, the classification unit 252 classifies a change in adisplay rack as one of a plurality of types, and therefore theevaluation unit 253 can make an evaluation using a thus classifiedresult, in the image processing device 250 according to the presentexample embodiment. Accordingly, the evaluation unit 253 can accuratelyevaluate a state of the display rack.

Further, in each of the aforementioned example embodiments, a capturedimage captured by the image capturing device 2 may be, for example, acaptured image in which an image of goods goods piled up on a wagon iscaptured. The image processing device can detect a change area bycomparing the captured image in which an image of the goods goods piledup on the wagon is captured with a background image. Accordingly, theimage processing device according to each of the example embodiments ofthe present disclosure may use captured images in which images of goodsgoods displayed in various display methods are captured, without beinglimited to a display rack on which goods goods are displayed in such away that all the faces of the goods goods are visible.

(Hardware Configuration)

Each component in each device according to each example embodiment ofthe present disclosure represents a function-based block. For example, apart or the whole of each component in each device is provided by anarbitrary combination of an information processing device 900 and aprogram, as illustrated in FIG. 27. FIG. 27 is a block diagramillustrating an example of a hardware configuration of the informationprocessing device 900 providing each component in each device. Anexample of the information processing device 900 includes the followingconfiguration.

-   -   A central processing unit (CPU) 901    -   A read only memory (ROM) 902    -   A random access memory (RAM) 903    -   A program 904 loaded on the RAM 903    -   A storage device 905 storing the program 904    -   A drive device 907 for reading and writing of a recording medium        906    -   A communication interface 908 connected to a communication        network 909    -   An input-output interface 910 inputting and outputting data    -   A bus 911 connecting each component

Each component in each device according to each example embodiment isprovided by the CPU 901 acquiring and executing the program 904providing the function of the component. For example, the program 904providing the function of each component in each device is previouslystored in the storage device 905 or the ROM 902, and is loaded onto theRAM 903 and executed by the CPU 901 as needed. The program 904 may beprovided for the CPU 901 through the communication network 909, or maybe previously stored in the recording medium 906, be read by the drivedevice 907, and be provided for the CPU 901.

There are various modified examples of a method of providing eachdevice. For example, each device may be provided by an arbitrarycombination of an information processing device 900 and a program, thecombination being separate for each component. Further, a plurality ofcomponents included in each device may be provided by an arbitrarycombination of a single information processing device 900 and a program.

Further, a part or the whole of each component in each device isprovided by another general-purpose or dedicated circuit, a processor,or the like, or a combination thereof. The above may be configured witha single chip or may be configured with a plurality of chips connectedthrough a bus.

A part or the whole of each component in each device may be provided bya combination of the aforementioned circuit or the like, and a program.

When a part or the whole of each component in each device is provided bya plurality of information processing devices, circuits, or the like,the plurality of information processing devices, circuits, or the likemay be arranged in a concentrated manner or be arranged in a distributedmanner. For example, the respective information processing devices,circuits, or the like may be provided in a form of being connected withone another through a communication network such as a client-serversystem or a cloud computing system.

The respective aforementioned example embodiments are preferred exampleembodiments of the present disclosure, and the scope of the presentdisclosure is not limited to the respective aforementioned exampleembodiments; and a person skilled in the art may make exampleembodiments which include various changes through modifying andsubstituting the respective aforementioned example embodiments withoutdeparting from the spirit and scope of the present disclosure.

The whole or part of the example embodiments disclosed above can bedescribed as, but not limited to, the following supplementary notes.

(Supplementary Note 1)

An image processing device comprising:

a detection means configured to detect a change area related to adisplay rack from a captured image in which an image of the display rackis captured;

a classification means configured to classify a change related to thedisplay rack in the change area; and

an evaluation means configured to evaluate a display state of goods,based on a classification result.

(Supplementary Note 2)

The image processing device according to supplementary note 1, whereinthe evaluation means calculates an amount of display of the goods, basedon the classification result, information about the change area, andmonitored area information indicating a target area where the displaystate of the goods is monitored in the captured image.

(Supplementary Note 3)

The image processing device according to supplementary note 2, wherein

the evaluation means evaluates the display state of the goods, based ona transition of the amount of display.

(Supplementary Note 4)

The image processing device according to any one of supplementary notes1 to 3, further comprising

an output control means configured to output information about thedisplay state of the goods to an output device, based on an evaluationresult by the evaluation means.

(Supplementary Note 5)

The image processing device according to any one of supplementary notes1 to 4, wherein

the classification means classifies the change related to the displayrack in the change area, based on a previously learned model of thechange related to the display rack or distance information indicating animage captured before an image capturing time of the captured image.

(Supplementary Note 6)

The image processing device according to supplementary note 5, wherein

the captured image is a color image,

the detection means detects the change area by comparing the capturedimage with background information indicating the image captured beforethe image capturing time of the captured image, and

the classification means classifies the change related to the displayrack in the change area, based on the previously learned model.

(Supplementary Note 7)

The image processing device according to supplementary note 5, wherein

the captured image includes a first captured image being a color imageand a second captured image being a distance image in which an image ofan image capturing range of the first captured image is captured withina predetermined time from the image capturing time of the first capturedimage,

the detection means detects the change area by comparing the firstcaptured image with background information indicating the image capturedbefore the image capturing time of the first captured image, and

the classification means classifies the change related to the displayrack, based on a comparison result between a value of each pixelincluded in the second captured image and a value of each pixel in thedistance information captured before the image capturing time of thesecond captured image.

(Supplementary Note 8)

The image processing device according to supplementary note 5, wherein

the captured image is a distance image,

the detection means detects the change area by comparing the capturedimage with background information indicating the image captured beforethe image capturing time of the captured image, and

the classification means classifies the change related to the displayrack, based on a comparison result between a value of each pixelincluded in the captured image and a value of each pixel in the distanceinformation.

(Supplementary Note 9)

The image processing device according to supplementary note 7 or 8,wherein

the classification means classifies the change related to the displayrack in the change area as a type related to a change in goods displayedon the display rack or a type related to a change due to an object otherthan goods displayed on the display rack, based on the comparison resultbetween the value of each pixel included in the captured image being thedistance image, and a distance between the image capturing device andthe display rack.

(Supplementary Note 10)

The image processing device according to any one of supplementary notes6 to 9, wherein

the detection means tracks the change area between a plurality of thecaptured images, and

the classification means classifies the change related to the displayrack, based on a tracking result.

(Supplementary Note 11)

The image processing device according to supplementary note 10, wherein

the detection means includes a background information update meansconfigured to update the background information, based on the capturedimage used when detecting the change area, and the backgroundinformation, and,

when the tracking result of the change area satisfies a firstpredetermined condition, the background information update means updatesan area other than an area corresponding to the change area, in thebackground information.

(Supplementary Note 12)

The image processing device according to supplementary note 11, wherein,

when the tracking result of the change area satisfies a secondpredetermined condition, the background information update means updatesthe area corresponding to the change area, in the backgroundinformation.

(Supplementary Note 13)

The image processing device according to any one of supplementary notes5 to 12, wherein,

based on the change area and rack area information indicating a rackarea where the goods is displayed in the display rack, the detectionmeans determines whether or not the change area is include in aplurality of the rack areas, and,

when the change area is included in one of the rack areas, theclassification means classifies the change related to the display rackin the change area as a type related to a change in goods displayed onthe display rack.

(Supplementary Note 14)

An image processing method comprising:

detecting a change area related to a display rack from a captured imagein which an image of the display rack is captured;

classifying a change related to the display rack in the change area; and

evaluating a display state of goods, based on a classification result.

(Supplementary Note 15)

The image processing method according to supplementary note 14, furthercomprising

calculating an amount of display of goods, based on the classificationresult, information about the change area, and monitored areainformation indicating a target area where a display state of goods ismonitored in the captured image.

(Supplementary Note 16)

A computer-readable non-transitory recording medium recorded with aprogram causing a computer to execute:

detection processing of detecting a change area related to a displayrack from a captured image in which an image of the display rack iscaptured;

classification processing of classifying a change related to the displayrack in the change area; and

evaluation processing of evaluating a display state of goods, based on aclassification result.

(Supplementary Note 17)

The recording medium according to supplementary note 16, wherein

the evaluation processing calculates an amount of display of the goods,based on the classification result, information about the change area,and monitored area information indicating a target area where thedisplay state of the goods is monitored in the captured image.

REFERENCE SIGNS LIST

-   1 Goods monitoring system-   2 Image capturing device-   3 Output device-   4 Display rack-   10 Classification device-   100 Image processing device-   110 Acquisition unit-   120 Detection unit-   121 Foreground area detection unit-   123 Background information update unit-   130 Classification unit-   131 First extraction unit-   132 Second extraction unit-   134 Area change classification unit-   140 First storage unit-   141 Background information-   142 Rack change model-   150 Evaluation unit-   151 Display state update unit-   153 Calculation unit-   160 Output control unit-   170 Second storage unit-   171 Display state information-   172 Monitored area information-   221 Foreground area detection unit-   223 Background information update unit-   224 Foreground area tracking unit-   243 Foreground area information-   311 First acquisition unit-   312 Second acquisition unit-   331 First extraction unit-   332 Second extraction unit-   334 Area change classification unit-   335 Distance information update unit-   344 Distance information-   435 Distance information update unit-   521 Foreground area detection unit-   523 Background information update unit-   531 First extraction unit-   532 Second extraction unit-   541 Background information

What is claimed is:
 1. An image processing device comprising a processorconfigured to: detect a change area related to a display rack from acaptured image in which an image of the display rack is captured;classify a change related to the display rack in the change area; andevaluate a display state of goods, based on a classification result,wherein the captured image is a color image, the processor detects thechange area by comparing the captured image with background informationindicating the image captured before the image capturing time of thecaptured image, and the processor classifies the change related to thedisplay rack in the change area, based on a previously learned model ofthe change related to the display rack.
 2. The image processing deviceaccording to claim 1, wherein the processor calculates an amount ofdisplay of the goods, based on the classification result, informationabout the change area, and monitored area information indicating atarget area where the display state of the goods is monitored in thecaptured image.
 3. The image processing device according to claim 2,wherein the processor evaluates the display state of the goods, based ona transition of the amount of display over time.
 4. The image processingdevice according to claim 1, the processor further configured to outputinformation about the display state of the goods to an output device,based on an evaluation result.
 5. The image processing device accordingto claim 1, wherein the processor tracks the change area between aplurality of the captured images, and the processor classifies thechange related to the display rack, based on a tracking result.
 6. Theimage processing device according to claim 5, wherein the processorupdates the background information, based on the captured image usedwhen detecting the change area, and the background information, and,when the tracking result of the change area satisfies a firstpredetermined condition, the processor updates an area other than anarea corresponding to the change area, in the background information. 7.The image processing device according to claim 6, wherein, when thetracking result of the change area satisfies a second predeterminedcondition, the processor updates the area corresponding to the changearea, in the background information.
 8. The image processing deviceaccording to claim 1, wherein, based on the change area and rack areainformation indicating a rack area where the goods is displayed in thedisplay rack, the processor determines whether or not the change area isincluded in a plurality of the rack areas, and, when the change area isincluded in one of the rack areas, the processor classifies the changerelated to the display rack in the change area as a type related to achange in goods displayed on the display rack.
 9. An image processingmethod comprising: detecting a change area related to a display rackfrom a captured image in which an image of the display rack is captured;classifying a change related to the display rack in the change area; andevaluating a display state of goods, based on a classification result,wherein the captured image is a color image, the detecting comprisesdetecting the change area by comparing the captured image withbackground information indicating the image captured before the imagecapturing time of the captured image, and the classifying comprisesclassifying the change related to the display rack in the change area,based on a previously learned model of the change related to the displayrack.
 10. The image processing method according to claim 9, furthercomprising calculating an amount of display of goods, based on theclassification result, information about the change area, and monitoredarea information indicating a target area where a display state of goodsis monitored in the captured image.
 11. A computer-readablenon-transitory recording medium recorded with a program causing acomputer to execute: detection processing of detecting a change arearelated to a display rack from a captured image in which an image of thedisplay rack is captured; classification processing of classifying achange related to the display rack in the change area; and evaluationprocessing of evaluating a display state of goods, based on aclassification result, wherein the captured image is a color image, thedetection processing of detecting the change area by comparing thecaptured image with background information indicating the image capturedbefore the image capturing time of the captured image, and theclassification processing of classifying the change related to thedisplay rack in the change area, based on a previously learned model ofthe change related to the display rack.
 12. The recording mediumaccording to claim 11, wherein the evaluation processing calculates anamount of display of the goods, based on the classification result,information about the change area, and monitored area informationindicating a target area where the display state of the goods ismonitored in the captured image.